Skip to content

[chore](conf) Set enable_advance_next_id=true by default (#44790)#46425

Merged
gavinchou merged 1 commit intoapache:branch-3.0from
gavinchou:gavin-advance-id-by-default
Jan 5, 2025
Merged

[chore](conf) Set enable_advance_next_id=true by default (#44790)#46425
gavinchou merged 1 commit intoapache:branch-3.0from
gavinchou:gavin-advance-id-by-default

Conversation

@gavinchou
Copy link
Contributor

@gavinchou gavinchou commented Jan 5, 2025

pick #44790

Bind the generation of next_id to physical time to ensure its generation is monotonically increasing, even if we directly overwrite the FE metadata and then restart. This way, it supports a lossy cluster rollback in scenarios where major version upgrades are incompatible.

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@gavinchou
[chore](conf) Set enable_advance_next_id=true by default (apache#44790)
6eaeedd 
Bind the generation of next_id to physical time to ensure its generation
is monotonically increasing, even if we directly overwrite the FE
metadata and then restart. This way, it supports a lossy cluster
rollback in scenarios where major version upgrades are incompatible.
@Thearas
Copy link
Contributor

Thearas commented Jan 5, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@gavinchou
Copy link
Contributor Author

gavinchou commented Jan 5, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 5, 2025

TPC-H: Total hot run time: 40736 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6eaeedd416b5d724e7c1e4ba1c1da32587c7b334, data reload: false

------ Round 1 ----------------------------------
q1	17568	7287	7225	7225
q2	2065	194	178	178
q3	10634	1075	1133	1075
q4	10562	725	754	725
q5	7742	2836	2801	2801
q6	238	155	149	149
q7	995	617	593	593
q8	9384	1929	2028	1929
q9	6678	6411	6358	6358
q10	7151	2339	2274	2274
q11	480	280	267	267
q12	417	218	207	207
q13	17762	2990	3006	2990
q14	238	213	212	212
q15	567	533	524	524
q16	679	618	605	605
q17	968	595	572	572
q18	7191	6632	6741	6632
q19	1387	1129	1106	1106
q20	472	202	196	196
q21	4092	3162	3250	3162
q22	1107	1011	956	956
Total cold run time: 108377 ms
Total hot run time: 40736 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7279	7214	7244	7214
q2	330	232	227	227
q3	2890	2950	2926	2926
q4	2030	1836	1880	1836
q5	5736	5743	5749	5743
q6	226	144	141	141
q7	2268	1787	1822	1787
q8	3350	3505	3510	3505
q9	8765	8867	8867	8867
q10	3550	3573	3526	3526
q11	601	503	502	502
q12	792	599	601	599
q13	9813	3207	3173	3173
q14	317	281	271	271
q15	578	521	539	521
q16	728	677	660	660
q17	1854	1613	1623	1613
q18	8362	7720	7800	7720
q19	1651	1602	1596	1596
q20	2073	1837	1885	1837
q21	5561	5231	5388	5231
q22	1139	1082	1008	1008
Total cold run time: 69893 ms
Total hot run time: 60503 ms

@doris-robot
Copy link

doris-robot commented Jan 5, 2025

TPC-DS: Total hot run time: 197690 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6eaeedd416b5d724e7c1e4ba1c1da32587c7b334, data reload: false

query1	1290	904	884	884
query2	6235	2097	2057	2057
query3	10837	4470	4121	4121
query4	62170	30205	23462	23462
query5	5180	452	452	452
query6	401	173	179	173
query7	5473	308	300	300
query8	310	244	241	241
query9	8452	2682	2682	2682
query10	451	262	276	262
query11	17378	15292	15929	15292
query12	168	107	107	107
query13	1406	469	456	456
query14	10626	6989	7070	6989
query15	211	181	178	178
query16	6965	498	478	478
query17	1181	601	583	583
query18	1594	326	324	324
query19	201	159	159	159
query20	122	113	109	109
query21	208	104	108	104
query22	4588	4360	4843	4360
query23	34625	34374	34299	34299
query24	6261	2933	2960	2933
query25	546	430	434	430
query26	647	179	182	179
query27	1783	300	307	300
query28	4377	2497	2471	2471
query29	742	481	483	481
query30	246	166	164	164
query31	1042	817	853	817
query32	67	57	55	55
query33	395	275	283	275
query34	932	512	538	512
query35	848	734	733	733
query36	1092	963	968	963
query37	125	72	72	72
query38	4167	4197	4049	4049
query39	1518	1511	1474	1474
query40	203	97	98	97
query41	50	50	50	50
query42	120	104	100	100
query43	535	492	489	489
query44	1172	833	841	833
query45	186	180	178	178
query46	1158	743	745	743
query47	2068	1927	1921	1921
query48	487	383	386	383
query49	754	387	389	387
query50	845	432	415	415
query51	7511	7294	7341	7294
query52	104	91	90	90
query53	259	182	194	182
query54	615	479	472	472
query55	77	75	76	75
query56	265	263	246	246
query57	1301	1184	1173	1173
query58	221	208	214	208
query59	3300	2965	2885	2885
query60	275	255	248	248
query61	107	110	112	110
query62	772	704	656	656
query63	221	192	188	188
query64	1334	703	650	650
query65	3282	3190	3252	3190
query66	691	300	306	300
query67	16041	15593	15556	15556
query68	4105	573	564	564
query69	424	267	266	266
query70	1139	1122	1153	1122
query71	377	275	260	260
query72	6398	4227	4036	4036
query73	754	351	349	349
query74	10219	8931	8912	8912
query75	3320	2640	2698	2640
query76	1783	1085	1151	1085
query77	490	278	282	278
query78	10657	9600	9490	9490
query79	1189	584	604	584
query80	833	439	444	439
query81	521	241	240	240
query82	1358	119	116	116
query83	250	145	148	145
query84	286	83	76	76
query85	902	315	298	298
query86	320	311	281	281
query87	4538	4387	4300	4300
query88	3546	2435	2385	2385
query89	414	296	288	288
query90	1988	184	191	184
query91	185	151	149	149
query92	61	51	52	51
query93	1330	535	538	535
query94	761	303	292	292
query95	360	262	254	254
query96	613	282	284	282
query97	3314	3210	3195	3195
query98	221	207	196	196
query99	1940	1299	1302	1299
Total cold run time: 313410 ms
Total hot run time: 197690 ms

@doris-robot
Copy link

doris-robot commented Jan 5, 2025

ClickBench: Total hot run time: 33.24 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6eaeedd416b5d724e7c1e4ba1c1da32587c7b334, data reload: false

query1	0.04	0.03	0.04
query2	0.07	0.03	0.03
query3	0.24	0.07	0.06
query4	1.63	0.10	0.10
query5	0.53	0.52	0.51
query6	1.13	0.73	0.73
query7	0.03	0.02	0.01
query8	0.04	0.05	0.04
query9	0.58	0.50	0.52
query10	0.55	0.55	0.57
query11	0.14	0.10	0.10
query12	0.14	0.12	0.11
query13	0.62	0.61	0.59
query14	2.95	2.97	2.94
query15	0.89	0.83	0.82
query16	0.38	0.38	0.38
query17	1.04	1.04	1.04
query18	0.23	0.21	0.22
query19	1.91	1.90	1.95
query20	0.01	0.01	0.02
query21	15.35	0.57	0.57
query22	2.57	2.84	1.37
query23	16.82	1.00	0.82
query24	3.04	1.50	2.50
query25	0.15	0.24	0.12
query26	0.63	0.13	0.14
query27	0.05	0.05	0.03
query28	9.20	1.09	1.08
query29	12.60	3.23	3.22
query30	0.24	0.06	0.06
query31	2.87	0.40	0.38
query32	3.23	0.46	0.47
query33	2.99	3.07	3.06
query34	17.22	4.46	4.48
query35	4.55	4.52	4.57
query36	0.66	0.51	0.49
query37	0.09	0.06	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.03
query40	0.16	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 105.8 s
Total hot run time: 33.24 s

@gavinchou gavinchou merged commit 71a3cc7 into apache:branch-3.0 Jan 5, 2025
21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@gavinchou @Thearas @doris-robot